Skip to content

Add parallel setup driver for GCP resources - #67

Merged
shlbatra merged 1 commit into
mainfrom
feat/parallel-setup-resources
Aug 7, 2026
Merged

Add parallel setup driver for GCP resources#67
shlbatra merged 1 commit into
mainfrom
feat/parallel-setup-resources

Conversation

@shlbatra

@shlbatra shlbatra commented Aug 7, 2026

Copy link
Copy Markdown
Owner

Adds scripts/setup_resources.sh, which runs the three setup scripts concurrently and reports a status table. Wall clock becomes Composer's ~25 min instead of the sum of all three.

Mirrors destroy_resources.sh in flags and structure, so the create/destroy pair stays symmetric.

Usage

./scripts/setup_resources.sh                        # all three in parallel
./scripts/setup_resources.sh --skip-composer        # skip the slow one
./scripts/setup_resources.sh --feature-store-args="fraud fraud_user"
./scripts/setup_resources.sh --serial               # one at a time, for debugging
./scripts/setup_resources.sh --dry-run

Output

=== Setup Summary ===
RESOURCE             STATUS     DURATION   LOG
--------             ------     --------   ---
composer             SUCCESS    24m18s     .setup-logs/20260807-135736/composer.log
feature-store        FAILED(7)  0m02s      .setup-logs/20260807-135736/feature-store.log
artifact-registry    SUCCESS    0m41s      .setup-logs/20260807-135736/artifact-registry.log

Exit code is 0 only if every target that actually ran succeeded.

Design notes

  • Per-script log files. Three interleaved streams of gcloud output on one terminal is unreadable. Failed logs are tailed automatically in the summary.
  • Deliberately not set -e. A failing child script has to be recorded and reported next to the others, not abort the driver and hide the rest.
  • Pre-flight checks verify gcloud auth and that feature_store is importable. Without them, a missing venv fails feature-store one second in while Composer keeps building for 25 minutes.
  • Skipped targets appear in the summary. A table that silently omits them reads as "all three succeeded" when only one ran.
  • Accurate per-task durations — each task's finish time is stamped when detected (2s poll), rather than reading the clock after the whole loop exits, which would give every task the slowest one's duration.
  • bash 3.2 compatible (macOS /bin/bash): indexed arrays only, no wait -n, no associative arrays.
  • --timeout kills the process group rather than the parent, since gcloud spawns children that would otherwise outlive it. Off by default — interrupting a half-created Composer environment is worse than waiting.

Testing

Exercised against stub scripts covering: all-success, partial failure with exit-code propagation, skip reporting, serial mode, timeout watchdog, dry-run, unknown-argument rejection, all-skipped, and pre-flight pass. Not yet run against live GCP.

🤖 Generated with Claude Code

Runs setup_composer.sh, setup_feature_store.sh and setup_artifact_registry.sh
concurrently and prints a status table. Wall clock becomes Composer's ~25 min
rather than the sum of all three.

Mirrors destroy_resources.sh in flags and structure:
  --skip-composer / --skip-feature-store / --skip-artifact-registry
  --serial, --dry-run, --timeout, --heartbeat, --log-dir

Details:
- Each script writes to its own log file; three interleaved gcloud streams on
  one terminal are unreadable. Failed logs are tailed in the summary.
- Not `set -e` — a failing child must be recorded and reported alongside the
  others, not abort the driver and hide the rest. Exit 1 if any target failed.
- Pre-flight checks gcloud auth and that feature_store is importable, so a
  missing venv fails in one second rather than 25 minutes into a Composer build.
- Skipped targets appear in the summary; omitting them would read as "all
  succeeded" when only one ran.
- bash 3.2 compatible (macOS /bin/bash): indexed arrays, no `wait -n`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@shlbatra
shlbatra merged commit f90acf7 into main Aug 7, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant